Week 6: Final Project

Background

Our assignment for the final lens was to create a lens using any modality (or combination of modalities) intended for one of the following applications: Utility/Tools, Games, Education, Commerce (Advertising, Fashion, Shopping/Retail, etc.), Storytelling, or Artistic Expression.

When I started to work on this lens, I was inspired by recently having to dress up in “Hogwarts in Manhattan” theme for an in-house competition at my gym. I was a big Harry Potter nerd growing up, so I had no problem digging out my old Gryffindor robe and Harry Potter glasses, even though I’m technically a Ravenclaw.

However, because I chose to work in the world of Harry Potter, one of my first concerns in building a lens was taking into consideration what already has been done in Snap and/or AR. Some previous applications in Snapchat included virtual Hogwart house outfits for Warner Bros' new Tournament of Houses series, Quidditch-themed Bitmoji, and a shoppable mini Hogwarts Mystery game where you could practice spells by drawing on your screen.

The Quidditch-themed Bitmoji lens in Snapchat released on July 31, 2018.

A big draw of augmented reality for me is being able to converge the worlds of fantasy and reality. In the same way that you could cosplay characters, drink butterbeer (I even made it once myself), and get a wand at the Ollivanders in Universal Studios, it’s fun to have these elements of fantasy and magic in your “real” life.  

So as I approach this final project, I considered using the world/environment modality and applying this lens as part of an experience at a Harry Potter store or theme park.

Through the Lens (of a Seeker)

I created a lens based on the Golden Snitch (also just known as the Snitch), a small golden ball that would fly (and sometimes pause or hover) around the field in a game of Quidditch. A team’s Seeker has to catch the snitch in order to end the match. In Lens Studio, I worked on mimicking the Golden Snitch’s movement by using move and rotation tweening to animate the object. I also experimented with VFX including confetti particles that could possibly be triggered by “catching” the snitch.

A screenshot of the lens in progress in Lens Studio.

A demo of the Seeker lens in action. Since this test, I adjusted the movement of the snitch to “fly” a little further off the screen, but this video gives you a sense of the animation.

Additionally, I was thinking about more applications of this playful world lens in a real place where the user could make more magical discoveries. I added image tracking to a blank folded piece of parchment paper. By moving your lens over the paper (come back to this image after you load the lens in Snapchat), the Marauder's Map – a magical map of Hogwarts – would reveal itself.

GIF of the Marauder’s Map animation.

Next Steps

There are many more next steps and possibilities for this lens. It would be great to continue building this lens out with scripted interactions (getting close to the snitch or catching it), adding audio components and triggers, having a front camera face lens component, and mapping objects to custom landmarks (back to the idea of applying this lens to Harry Potter World).

Week 5: Environment

During Week 5, we explored the environment modality and created world lenses in Lens Studio. Our assignment for the week: create a lens that encourages users to go outside and interact with their environment in some way.

Winter in New York City has been unusually warm lately and with barely any snowfall. I truly enjoy seeing the city when it’s covered in a blanket of snow (and I’m known for venturing out in snowstorms to document photos and videos), so the weather this past season has been disappointing. This very disappointment inspired me to create a lens that would give you the sense of being in a snowy world.

I started by playing around with a snow texture for ground segmentation and mixing it up with snow particles and the ice post effect. While playing around with the snow texture, I accidentally covered up half of the screen so that the view appeared to be deep within the snow – but I actually liked the effect and decided to keep it. What would you do if you were buried in snow? That made me think of adding objects like a snowboard, but after presenting this concept, I realized that foot-tracking a snowboard that is usually sideways and strapped to your feet didn’t make the most sense. I decided to try out the foot tracking feature with snow shoes (more logical) instead.

A screenshot of the 3D scene with the default foot mesh inside the snowshoes.

Trying out foot tracking was a learning experience with understanding how to properly attach the objects to the foot mesh and occlude the feet – and I had some glitchy looking outcomes. Ultimately, I decided it was best to only track the snowshoes to your feet instead of making them appear like they’re on your feet (they just didn’t look right).

Occluded feet inside the snowshoes not looking that realistic.

The final scene in Lens Studio.

A demo of this “Snowed In” lens in action while walking outside.

Week 4: Body

Our assignment for Week 5: Create a lens that utilizes the moving body to trigger digital content.

A few weeks ago, a few friends and I talked about learning how to dance the chorus of NewJeans’ “OMG” (a major earworm, by the way) so that we could do the dance together at some point. Since then, I’ve been sent so many video memes about the dance or other people performing it, and just felt like I absolutely had to make a reactive body lens to this song for this assignment.

A video of NewJeans themselves rehearsing “OMG” – in case you may be inspired to practice it yourself.

I wanted to understand how body tracking and triggers worked in Lens Studio, so I opened up both the Full Body Triggers template and the Body Segmentation template and started going through the Inspector panel for the various objects and effects. The possibilities feel unlimited here.

Exploring the Full Body Tracking Controller and attached objects through the Full Body Triggers template.

Ultimately, I wanted to create a lens that could be synced to the audio of the chorus. I used the Timed Effect example in the Body Segmentation template and modified the controller script and effects to time the different scene objects to the music. I also added an “OMG” graphic that’s triggered by moving the hand above the head (with an invisible “button”), which should appear while doing the signature “OMG” dance move.

A screenshot of the Controller Script for cycling through each effect that I timed to the music.

A snippet of the “OMG” lens in action.

My own attempt at the dance for a quick lens demo.

Content Link Block

Select a page and create a visual link to it. Learn more

ITP Blog

Week 3: Image

Our assignment for Week 3: Tell a story across AR image targets. Pick a piece of print media (magazine, newspaper, book, something you’ve created). Choose at least 2 images within it (up to 4) and augment them to create a narrative.

I chose to work with the Approval Matrix printed in New York Magazine, which is their “guide to what’s highbrow, lowbrow, brilliant, and despicable.” Because everything selected is summarized down to a brief point, I often don’t always know the reference or the full picture (pun intended) of the point on the matrix. This is where I thought the Approval Matrix could benefit from some additional images (or text) through AR to add to the storytelling.

Original Approval Matrix Image

Below is New York Magazine’s Approval Matrix from the Week of February 13. It is published online and in the printed magazine. This image can be used for testing the lens.

Lens Studio AR

I chose three items in the Highbrow/Brilliant quadrant to work with: famous whip performer Jack Lepiarz (didn’t know who he was), LeBron James breaking the all-time scoring record in the NBA, and the Copland Dance Episodes at the New York City Ballet. I started with the Copland Dance Episodes by experimenting with overlaying a GIF I made from a video preview of the New York City Ballet performance over the original photograph, and for Jack Lepiarz, I found a TikTok clip of one of his performances.

Preview

For the LeBron James point on the Approval Matrix, I overlayed a clip of him making the all-time scoring record basket. I experienced some issues publishing this lens; I will update this page with the Snap lens link when it’s available.

Week 2: Face

Our second assignment: Pick an adjective and create a face lens to depict its meaning.

For this assignment, I picked “whimsical” as my adjective. Whimsical means “playfully quaint or fanciful, especially in an appealing and amusing way” and I decided to go for a filter that felt like being at a garden tea party, à la Alice in Wonderland. I was browsing the Asset Library and came across a 3D stack of plates, bowls, and cups in the Kitchen Pack and thought they could make a great little hat.

I also wanted to use this first exercise in Lens Studio to play with some of the face effects like Liquify (to enlarge the eyes and give them a bit of a cartoonish look), Retouch (to soften skin and whiten the eyes and teeth), and the Makeup Controller (to add blush, lipstick, and eyebrow fill) from one of the templates.

The interesting thing I did learn after adding the flowers is that the 3D objects appear differently in the Preview screen versus previewing the lens in the Snapchat app. I wanted the face to appear within the flowers like the preview window in this first screenshot, but they ended up being quite tiny on my phone.

I then scaled the objects to the correct size that I wanted them to appear within the Snapchat app, but as you can see in the preview window in Lens Studio in the following screenshot, the flowers appear offscreen.

The video below is the final test of the lens preview in Snapchat.

Week 1: LoFi AR

Our first assignment: Using no digital technology, create an AR experience.

I wanted to take paper cut-out digital art back to its original LoFi version. I cut a few pieces of paper to create the effect and glued them together to create colorful frame for viewing reality. In this photo, my subject was a Lucky Cat that sits at a coworker’s desk.